Systems and methods for improving recommendation systems

ABSTRACT

Data about clients, items included in an item inventory, executable code for recommendation processes, and executable code for feature selection process(es), are stored. The recommendation processes are used to identify items in the item inventory that are recommended for specific clients, and the feature selection process(es) is/are used to select features that are displayed to entities tasked with selecting items from the item inventory for clients. Selection feedback and client feedback are received. One or more feature selection process(es) is/are modified based on the selection feedback and/or the client feedback, to thereby change which, where or/or how certain portion of the data about clients is displayed to the entities tasked with selecting items from the item inventory for the clients, and/or change which, where and/or how certain portion of the data about items is displayed to the entities tasked with selecting items from the item inventory for the clients.

CROSS REFERENCE TO OTHER APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/241,899 entitled SYSTEMS AND METHODS FOR IMPROVING RECOMMENDATIONSYSTEMS filed Aug. 19, 2016 which is incorporated herein by referencefor all purposes.

BACKGROUND

Recommendations systems are becoming increasingly common in online salesand digital media sites. Such systems typically use information about auser's preferences and purchasing behavior to recommend content and/orgoods in which the user may be interested. Such recommendations systemsoften present a user with recommended content and/or goods without theuser having to explicitly request or look for the content and/or goods.In this way, content and/or goods of which the user may have beenunaware, may be brought to the attention of the user, for purchase.However, existing recommendation systems often recommend certain contentand/or goods that are inappropriate for and/or poorly matched to users.Accordingly, there is a need to improve upon existing recommendationsystems.

Where a recommendation system uses one or more processors to assist withselecting items for clients from a large item inventory, there is a hugeamount of data about the clients and items within the item inventorythat can be presented. If all of that data were to be presented, anentity tasked with making the final decision as to what items should beselected for a client would be so overwhelmed with data that the overallperformance of the recommendation system would be very poor. In otherwords, there is a big data problem related to the visualization of bigdata including client data and item inventory data.

SUMMARY

In accordance with an embodiment of the present technology, a systemincludes one or more processors, one or more data stores, and a feedbackinterface. The one or more data stores, e.g., databases, are adapted tostore client attribute data and client history data, for each of aplurality of clients, and item attribute data, for each of a pluralityof items included in an item inventory. Additionally, the data store(s)are adapted to store executable code for each of a plurality ofdifferent recommendation processes, wherein each of the differentrecommendation processes may utilize the client attribute data, theclient history data and/or the item attribute data in a different mannerthan the other recommendation processes. Also, the data store(s) areadapted to store the executable code for one or more feature selectionprocess(es). The executable code for the plurality of differentrecommendation are each executable by the processor(s) to identify, forany particular one of the clients, a subset of the items in the iteminventory that is recommended for the particular one of the clients. Theexecutable code for the one or more feature selection process(es) is/areexecutable by the processor(s) to select which item attributes and whichclient attributes are to be presented to entities the may be tasked withselecting items for the clients from the item inventory.

The processor(s) is/are also adapted to, for each client of two or moreof the clients, execute the executable code for two or more of theplurality of recommendation processes to thereby identify a plurality ofdifferent subsets of the items in the item inventory that arerecommended for the client. The processor(s) is/are also adapted toexecute the executable code for at least one of the feature selectionprocess(es) to thereby select features for presentation, to one of theentities that is tasked with selecting items from the item inventory forthe client. In accordance with specific embodiments, the selectedfeatures include information about two or more item attributesassociated with items includes in two or more of the different subsetsof the items in the inventory that are identified as a result of theexecutable code for two or more of the recommendation processes beingexecuted, and information about two or more of the client attributesassociated with the client.

The feedback interface is adapted to receive, for each client of two ormore of the clients, indications of specific ones of the items that areselected from the item inventory for the client by the one of theentities that is tasked with selecting items from the item inventory forthe client, and client feedback data about one or more of the itemsselected from the item inventory for the client by the one of theentities that is tasked with selecting items from the item inventory forthe client. The feedback interface is also adapted to update the storedclient history data based the client feedback data.

In accordance with certain embodiments, at least one of the featureselection process(es), that is/are used by the one or more processors toselect features for presentation, is modified based on the updatedstored client history data, to thereby improve performance of thesystem. The modifying of the feature selection process, based on theupdated client history information, can, for example, result inmodifying the information about item attributes and/or client attributesthat are selected for presentation being modified to thereby improveperformance of the system.

In accordance with specific embodiments, the system is adapted tosystematically change which features are selected by the featuresselection process for presentation at different instances that featuresare selected for presentation. Additionally, the feedback system isadapted to identify which of the features when presented result in theitems being selected by the entities for the clients receiving positiveclient feedback, and which of the features when presented result in theitems being selected by the entities for the clients receiving negativeclient feedback. Based thereon, the feature selection process can bemodified so that the features that are identified as resulting inpositive client feedback are thereafter selected for presentation or aremore prominently presented, and the features that are identified asresulting in negative client feedback are thereafter not selected forpresentation or are less prominently presented.

In accordance with certain embodiments, the processor(s) are alsoadapted to access the client history data to produce training tasks forthe entities that may be tasked with selecting items for the clientsfrom the item inventory. In such embodiments, the system can include atraining interface that is adapted to present at least some of thetraining tasks to at least some of the entities that may be tasked withselecting items for the clients from the item inventory, and providetraining feedback to the entities that are presented with the trainingtasks. The training feedback can provide indications, which are based onthe client history data, regarding whether particular ones of theclients, for which the entities selected items from the item inventoryin response to the training tasks, had provided positive or negativefeedback regarding the items.

The embodiments described herein can beneficially be used to analyze bigdata related to clients and item inventory, and select which subsets ofdata should be presented to provide for good overall system performance.Such embodiments also utilize feedback to improve the overall systemperformance over time. In such manners, embodiments of the presenttechnology address and at least partially overcome big data problemsrelated to the visualization of big data (including client data and iteminventory data) by a recommendation system.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system that provides recommendations to entitiesthat select items for clients from an item inventory.

FIG. 2 illustrates an example of the information that can be displayedto an entity tasked with selecting items from an item inventory for aclient.

FIG. 3 is a high level flow diagram used to summarize methods, accordingto embodiments of the present technology, for improving the performanceof a recommendation system.

FIG. 4 is a high level flow diagram that provides exemplary details asto how one or more feature selection process(es) can be modified toimprove the performance of a recommendation system, in accordance withspecific embodiments of the present technology.

DETAILED DESCRIPTION

As noted above, where a recommendation system uses one or moreprocessors to assist with selecting items for clients from a large iteminventory, there is a huge amount of data about the clients and itemswithin the item inventory that can be presented. If all of that datawere to be presented, an entity tasked with making the final decision asto what items should be selected for a client would be so overwhelmedwith data that the overall performance of the recommendation systemwould be very poor. In other words, there is a big data problem relatedto the visualization of big data including client data and iteminventory data, as noted above.

Embodiments of the present technology intelligently, and in someinstances systematically, analyze big data related to clients and iteminventory, and select which subsets of data should be presented toprovide for good overall system performance. Embodiments of the presenttechnology also utilize feedback to improve the overall systemperformance over time. In other words, embodiments of the presenttechnology address and at least partially overcome the above summarizedbig data problem related to the visualization of big data includingclient data and item inventory data.

In the following detailed description, reference is made to theaccompanying drawings that form a part hereof, and in which is shown byway of illustration specific illustrative embodiments. It is to beunderstood that other embodiments may be utilized and that logical,functional, mechanical and electrical changes may be made. The followingdetailed description is, therefore, not to be taken in a limiting sense.In the description that follows, like numerals or reference designatorswill be used to refer to like parts or elements throughout. In addition,the first digit of a reference number identifies the drawing in whichthe reference number first appears.

Most existing recommendation systems are computer implemented, and onceimplemented, rely entirely on computer implemented processes to providerecommendations to users. However, while computers are excellent atanalyzing structured data, computers are generally not very good atanalyzing unstructured data. Further, while computers are excellent atimplementing the processes that they are programmed to implement,computers are not very good at parsing through the results of multipleprocesses or appreciating that some processes provide better resultsthan others for specific users. Additionally, recommendation systemstypically provide recommendations directly to the people, also referredto as consumers or clients, that determine whether or not they wouldlike to purchase the recommended content and/or goods, wherein suchcontent and/or goods can more generally be referred to individually asan item or collectively as items. Embodiments of the present technology,which are described below, provide improvements over such existingrecommendation systems.

FIG. 1 illustrates a system 102 that provides recommendations toentities that select items for clients from an item inventory, ratherthan providing the recommendations directly to the clients thatdetermine whether or not they would like to accept (e.g., purchase) therecommended items. The system 102 (which can also be referred to as arecommendation system 102) is especially useful for assisting with tasksthat are best performed by a combination of machines (e.g., computers)and humans. More specifically, the system 102 is especially useful fortaking advantage of the fact that computers are typically betterequipped to analyze or otherwise handle structured data, while humansare typically better equipped to analyze or otherwise handleunstructured data. Explained another way, the system 102 provides amechanism to combine machine-learning processes with expert-humanjudgment, as will be appreciated from the description below.

Referring to FIG. 1, the system 102 is shown as including one or moreprocessor(s) 112 that is/are communicatively coupled to a clientdatabase 122-1, an inventory database 122-2, an entity database 122-3,and a recommendation and feature selection process database 122-4. Thedatabases 122-1, 122-2, 122-3 and 122-4 can be individually referencedas a database 122, or collectively referenced as databases 122. Moregenerally, the databases 122 are examples of types of data stores.Alternative types of data stores can be used instead or, or in additionto the databases 122, to store the types of data described herein. Alsoshown in FIG. 1 are blocks 132, which represent entities that select oneor more items for clients, and blocks 142, which represent the clientsfor which the item(s) is/are selected. Such entities 132 can be peoplethat have an expertise related to the items included in the iteminventory. FIG. 1 also shows a feedback interface 152 that acceptsvarious types of feedback, including client feedback, selection feedbackand coverage feedback, each of which is discussed in additional detailbelow. The feedback interface 152 is shown as being communicativelycoupled to the databases 122. As will be described in additional detailbelow, in accordance with certain embodiments, data stored within thevarious databases 122 can be updated based on the various types offeedback received by the feedback interface 152.

In accordance with an embodiment, the client database 122-1 is adaptedto store client attribute data and client history data, for each of aplurality of clients 142. Clients can be, e.g., people that haverequested that items be selected for them and provided (e.g., mailed,shipped, sent, transmitted or otherwise provided) to them. The inventorydatabase 122-2 (which can also be referred to as an item database) isadapted to store item attribute data, for each of a plurality of itemsincluded in an item inventory. The entity database 122-3 is adapted tostore entity attribute data, for each of the entities 132 that may betasked with selecting items from the item inventory for one or more ofthe clients 142. As will be described in additional detail below, inaccordance with certain embodiments, the entity attribute data, which isstored in the entity database 122-3 for each entity of at least some ofthe entities 132, includes training history data indicative of one ormore types of training provided to the entity and/or efficacy of thetraining provided to the entity.

Still referring to FIG. 1, the recommendation and feature selectionprocess database 122-4 is adapted to store executable code for each of aplurality of different recommendation processes that are each executableby one or more of the processor(s) 112 to identify, for any particularone of the clients 142, a subset of the items in the item inventory thatis recommended for the particular one of the clients 112. Therecommendation and feature selection process database 122-4 is alsoadapted to store executable code for one or more feature selectionprocess(es) that is/are executable by one or more of the processor(s)112 to select which item attributes and which client attributes are tobe presented to entities (e.g., 132) the may be tasked with selectingitems for one or more of the clients 142 from the item inventory. Inaccordance with certain embodiments of the present technology, theprocessor(s) 112 can use the recommendation processes and featureselection process(es) stored in the database 122-4 to identify itemsfrom the item inventory to recommend for clients. More specifically, theprocessor(s) 112 can be adapted to execute the executable code for twoor more of the plurality of recommendation processes stored in thedatabase 122-4 to thereby identify a plurality of different subsets ofthe items in the item inventory that are recommended for a client.Further, the processor(s) 112 can be adapted to execute the executablecode for at least one of the feature selection process(es) to therebyselect features for presentation, to one of the entities that is taskedwith selecting items from the item inventory for the client. Inaccordance with certain embodiments, the selected features can includeinformation about two or more item attributes associated with itemsincludes in two or more of the plurality of different subsets of theitems in the inventory that are identified as a result of the executablecode for two or more of the plurality of recommendation processes beingexecuted, as well as information about two or more of the clientattributes associated with the client for which items are beingselected. In other words, two or more recommendation processes and oneor more feature selection process(es) are used select which items, itemattributes and client attributes are displayed to an entity tasked withselected items for a client, as well as where and/or how such attributesare displayed.

The items in the item inventory, for which item attribute data is storedin the inventory database 122-2, can be, for example, women's apparel,with each item in the item inventory corresponding to a different pieceof women's apparel. For a more specific example, the items in the iteminventory can include pants, skirts, dresses, shirts, sweaters, jackets,bracelets, necklaces, scarfs, hats and/or the like. The item attributedata stored in the inventory database 122-2 can include, for each of theitems in the item inventory, a plurality of item attributes. Some of theitem attributes can be objective attributes, while other attributes canbe subjective attributes. Exemplary objective item attributes include:an item type (e.g., pant, skirt, dress, etc.), an item designer and/orbrand, an item size, an item color, an item stock keeping unit (SKU)number, an item pattern (e.g., solid, checked, pinstripe, etc.), an itemmaterial (e.g., cotton, wool, synthetic, etc.), and/or the like.Exemplary subjective item attributes include: good for tall women, goodfor short women, good for busty women, good for women in the State ofTexas, good for preppy women, good for women in their 20s, good forwomen in their 50s, good for women with a long torso, casual, dressy,good for tropical vacations, good for spring, good for winter, fits trueto size, runs small, runs large, and/or the like. Subjective itemattributes may also specify for which one or more body types (e.g.,straight, pear, spoon, hourglass, top hourglass, inverted triangle,oval, diamond, etc.) an item is appropriate. At least some of theobjective item attributes can be automatically entered into theinventory database 122-2, while other objective item attributes can bemanually entered into the inventory database 122-2. In accordance withan embodiment, the subjective item attributes are specified by entities(e.g., people) that have an expertise in the type of items included inthe inventory and are manually entered into the inventory database122-2. The item inventory can also specify how many units of each itemare presently in the item inventory. Additionally, the item inventorycan specify how many units of each item have been ordered, as well aswhen ordered items are expected to be received to replenish the iteminventory. Item attributes can also specify how often an item has beenprovided to clients, how often the item has been accepted (e.g.,purchased) by clients and/or how often the item, once provided toclients, has been accepted (e.g., purchased) by clients. Forclarification, not all item attributes are mutually exclusive. Forexample, an item may be both “good for tall” and “good for averageheight”. These are just some examples of the types of item attributesthat can be stored in the inventory database 122-2, which examples arenot intended to be all encompassing. At least some of the itemattributes, or more generally at least a portion of the stored iteminformation, for each of the items in the item inventory, is initiallyat least partly based on inputs accepted from an entity that is taskedwith specifying item attributes. For example, a web based user interfacecan be presented to an entity that enables them to enter objective andsubjective information about items, e.g., in response to questions orthe like.

The client attribute data stored in the client database 122-1 caninclude client attributes associated with each of a plurality of clients142. Some of the client attributes can be objective attributes, whileother attributes can be subjective attributes. Exemplary objectiveclient attributes include: date of birth and/or age, height, weight,waist size, chest size, bra size, inseam length, torso length, shirtsize, dress size, pant size, residential address, whether or not theclient is a mother, occupation, and/or the like. Client attribute datamay also include image data for a client, which enables a picture of theclient to be displayed to an entity that is tasked with selecting itemsto provide to the client. Exemplary subjective client attributesinclude: preferred style (e.g., classic, preppy, casual, bohemian,etc.), preferred fit (e.g., loose, fitted, tight, etc.), preferred colorscheme (e.g., bright, subdued), preferred colors (e.g., red, blue),disliked colors (e.g., teal, purple), preferred patterns (e.g., solids,stripes, floral), and/or the like. Subjective client attributes can alsospecify to what extent a client likes specific styles, such as, but notlimited to, casual, edgy, bohemian, classic, glamorous, preppy,romantic, and/or the like. Subjective client attributes can be binary(e.g., no or yes, or 0 or 1), or can have greater levels of granularity(e.g., a number between 0 and 5, with 5 indicating the greatestpreference). Subjective client attributes can also specify to whatextent a client likes specific designers and/or brands. Subjectiveclient attributes may also specify how much a client indicates they arewilling to spend on specific types of items. Another exemplary clientattribute is body type (e.g., straight, pear, spoon, hourglass, tophourglass, inverted triangle, oval, diamond, etc.). The client historydata associated with a particular client can also specify how long theparticular client has been a member of the recommendation serviceprovided by the system 102, how often items have been shipped to theclient, which items were purchased or otherwise accepted by the client,which items were returned or otherwise not accepted by the client,and/or the like, each of which are objective types of information. Theclient history data associated with a particular client may also specifyhow much a client has actually spent on specific types of items. Theseare just some examples of the types of client history data that can bestored in the client database 122-1, which examples are not intended tobe all encompassing. In an embodiment, at least some of the clientattributes, or more generally at least a portion of the stored clientattribute data, for each of the clients, is initially at least partlybased on inputs accepted from the client. For example, a web based userinterface can be presented to a client that enables the client to enterobjective and subjective information about themselves, e.g., in responseto questions or the like.

In certain embodiments, the processor(s) 112 is/are adapted to identifya plurality of different subsets of the items in the item inventory thatare recommended for any particular one of the clients by using all orsome (i.e., each of two or more) of the plurality of differentrecommendation processes (for which executable code is stored in thedatabase 122-4) to generate a different subset of the items in theinventory that is recommended for the particular one of the clients.Such identified subsets (or portions thereof) of the items in theinventory can be displayed to the specific entity 132 (e.g., a person)that is tasked with selecting items from the item inventory for theparticular one of the clients. Accordingly, the processor(s) 112 can beused to parse through data about a relatively large group of itemsincluded in an item inventory, and narrow the relatively large groupdown to a relatively smaller group, and thereafter a person can usetheir expert human judgment to select items from the relatively smallergroup to provide (e.g., mail, ship, send, transmit or otherwise provide)to a client.

Assume, for example, that a particular entity 132 is tasked withselecting five items to send to a particular client 142. Where thenumber of items included in an inventory is relatively large, and thereis a relatively large number of different recommendation processes (forwhich executable code is stored in the database 122-4), the entitytasked with selecting items from the item inventory for one of theclients may be overwhelmed. For example, assume that there are tenthousand different items in the item inventory, and that there are onehundred different recommendation processes for which executable code isstored in the database 122-4. Further, assume that each of the differentsubsets of the items (identified by the processor(s) 112, using one ofthe recommendation processes) includes six items. Each of the subsets ofitems need not be mutually exclusive, e.g., it is possible that twodifferent subsets of items, identified using two differentrecommendation processes, may be the same, but that is highly unlikely.However, there is a greater likelihood that an individual item may beincluded in more than one of the different subsets of items. Continuingwith this example, and assuming that every one of the one hundreddifferent recommendation processes is utilized by the processor(s) 112to identify a subset of six recommended items, this would result in onehundred different subsets of six items being identified by theprocessor(s) 112. In other words, in this example, information about sixhundred items may be displayed the entity 132 tasked with selectingitems from the item inventory for one of the clients 142. Whilenarrowing ten thousand items down to six hundred items may be helpful,it is still likely that the entity tasked with selecting items from theitem inventory for one of the clients would be overwhelmed, or would atleast ignore many of the suggested items. Certain embodiments describedherein can be used to better assist the entity (tasked with selectingitems from the item inventory for one of the clients) with making theirselections for a client. More generally, embodiments of the presenttechnology described herein can be used to provide improved techniquesfor managing, analyzing and presenting big data.

In accordance with an embodiment, each of the different recommendationprocesses for which executable code is included in the database 122-4has a corresponding different theme. The themes associated withrecommendation processes may or may not be related to specific clientattributes. Exemplary themes that relate to specific client attributesinclude: good for tall women, good for short women, good for bustywomen, good for women in the state of Texas, good for preppy women, goodfor women in their 20s, good for women in their 50s, good for women witha long torso, casual selections, dressy selections, good for tropicalvacations, good for spring, good for winter, good for women who likeclassic styles, good for women that like bohemian styles and/or thelike. As can be appreciated from this exemplary list of themes, some ofthe themes (corresponding to recommendation processes) can bedescriptors that are the same as or similar to descriptors of attributesassociated with items in the item inventory. In accordance with anembodiment, for at least some of the themes, each theme (correspondingto a recommendation process) indicates a common attribute associatedwith items selected using the recommendation process. For example, thecommon attribute associated with items selected using the theme “goodfor tall women” is that all the items selected using the recommendationprocess having that theme are believed to be good for women that are atleast 5 feet 8 inches. For another example, the common attributeassociated with items selected using the theme “good for women in theState of Texas” is that all the items selected using the recommendationprocess having that theme are believed to be good for women that live inthe State of Texas. Recommendation processes can be even more specificthan those exemplary processes discussed above, or two or morerecommendation processes can be compounded, e.g., to identify items thatare “good for tall women in the State of Texas” or “good for preppywomen in winter”. As processes, or combinations of processes become morespecific, there coverage and level of confidence will likely decrease,in which case heuristics can be used to control a minimum coverageand/or confidence threshold.

Other themes can be more general and not specifically related to clientattributes. For example, further exemplary themes of recommendationprocesses include: best sellers; highest probability of being purchased;highest profit margin; most similar to previously purchased items; leastsimilar to previously purchased items; best for casual events; best forbusiness events; good for fitting in; good for standing out; and/or thelike. These are just a few examples that are not intended to be allencompassing. The types of recommendation processes and the themesassociated therewith can vary greatly, depending upon the types of itemsin an item inventory and the clients for which items are being selected.Preferably, new recommendation processes can be added to therecommendation process database 122-1 without affecting and/or requiringany specific coordination with the remaining portions of the system 102.This allows new processes to be tested and trained as desired, andpoorly performing processes to be removed.

One or more recommendation processes can calculate a distance orsimilarity metric between specific attributes associated with a clientand specific attributes associated with items in an item inventory, witheach distance or similarity metric being assigned a weight. Differentrecommendation processes may focus on different attributes and/or mayweight distance or similarity metrics differently than one another.Additionally, or alternatively, different recommendation processes canuse different ways to calculate the distance or similarity metricsbetween specific attributes associated with a client and correspondingattributes associated with items in an item inventory.

In accordance with certain embodiments of the present technology, theprocesses stored in the database 122-4 (and more specifically, theprocesses whose executable code is stored in the database 122-4) arecomposable processes, which can be selected and assembled (i.e.,combined) in various different combinations to satisfy various differentobjectives. The composability of the processes stored in the database122-4 promotes reuse of the processes. A composable process, as the termis used herein, is a process that is both modular and stateless. Modularmeans that a process may cooperate with other processes (by using theoutput of another process as its input and/or providing its output as aninput to another process) and that the process may be swapped out andreplaced with another process. Stateless means that a process treatseach call or invocation as an independent transaction, unrelated toprevious calls or invocations. With composable processes, the result(i.e., output) of a process can be is passed as an argument of (i.e., aninput to) a next process, and the result of the last process can be theresult (i.e., output) of the whole.

In accordance with certain embodiments of the present technology, eachof the composable processes that is stored in the database 122-4 isdesignated as (e.g., tagged as) either being executable using on-demandprocessing, or being executable using batch-processing. In accordancewith certain embodiments of the present technology, each of thecomposable processes includes corresponding metadata and a correspondingcomputation function. The metadata for each of the composable processesthat is designated as being executable using on-demand processing caninclude at least a name of the composable process, an author of thecomposable process and a pointer to the corresponding computationfunction. The metadata for each of the composable processes that isdesignated as being executable using batch-processing can include atleast a name of the composable process, an author of the composableprocess and a pointer to a result of the composable process (whichresult can be stored as data in the batch-processing results data store,not shown). Each computation function can be a class, and allcomputation functions of composable processes that are designated asbeing executable using on-demand processing may inherit from a top levelbase class. In certain embodiments, each of the composable processes canbe thought of as a directed graph or tree, which may include a root aswell as some leaves.

In accordance with certain embodiments of the present technology, theformat of the output of each of the composable processes is independentof whether the composable process is, was, or will be processed usingon-demand or batch-processing. In other words, the outputs of all of thecomposable processes, whether they are designated as being executableusing on-demand processing or batch-processing, should have the sameformat. Beneficially, with such embodiments, when a composable processis being developed, e.g., by a data scientist, the data scientist (whichcan also be referred to as an process developer) need not think aboutwhether one or more other composable process(s) that the composableprocess being developed may invoke and/or be combined or composed withis/are designated for on-demand processing or batch-processing, becausethe structure of the output of each of the composable processes isindependent of whether it is, was, or will be processed using on-demandor batch-processing.

In accordance with an embodiment, at least one of the feature selectionprocess(es) may be used by the processor(s) 112 to determine, for eachof the recommendation processes having a theme that indicates a commonattribute associate with items selected using the recommendationprocess, whether or not any particular one of the clients satisfies thecommon attribute. Then, based on such determinations, the processor(s)112 can eliminate from being displayed, to an entity 132 that is taskedwith selecting items from the item inventory for the particular one ofthe clients 142, information about one or more of the subsets identifiedusing any recommendation process having a theme that indicates a commonattribute not satisfied by the particular one of the clients. Forexample, where a particular client is a women that is 5 feet 1 inchestall and lives in the State of Oregon, the processor(s) 112 candetermine that the particular client does not satisfy the attributes“tall” and “living in the State of Texas”. Accordingly, the processor(s)112 can eliminate from being displayed, to an entity 132 that is taskedwith selecting items from the item inventory for that particular client142, information about a subset of items believed to be good for tallwomen, and information about a subset of items believed to be good forwomen living in the State of Texas. One way that the processor(s) 112can do this is to not run the recommendation process having the theme“good for tall women”, and not run the recommendation process having thetheme “good for women living in the State of Texas”. Alternatively, theprocessor(s) 112 can executed the executable code for the recommendationprocesses having the themes “good for tall women” and “good for womenliving in the State of Texas”, but can ignore or hide the subsets ofitems identified using those processes, or rank or sort the subsets ofitems identified using those processes in such a manner that they willnot be displayed. Other variations are also possible and within thescope of an embodiment.

In accordance with an embodiment, the processor(s) 112 is/are adapted toselect for display, to an entity 132 that is tasked with selecting itemsfrom the item inventory for one of the clients 142, information abouttwo or more of the plurality of different subsets of the items in theinventory that are identified by running all or some of the processesincluded in the database 122-4. In order to accomplish this task, theprocessor(s) 112 can calculate, for each of at least some of the subsetsof items identified using recommendation processes, one or more metricsthat quantify one or more aspects of the subset of items. Exemplarymetrics that quantify aspects of a subset of items include, but are notlimit to, a metric indicative of probability of successfully (for makinga sale), a metric indicative of profit, a metric indicative ofsimilarity to previous items accepted by the client, a metric indicativeof dissimilarity to previous items accepted by the client, a metricindicative of a probability of client satisfaction, a metric indicativeof a probability of client retention, a metric indicative of optimallyutilizing a company's inventory, a metric indicative of a cost offulfilling a client's shipment and/or the like. The processor(s) 112 canthen use one or more processes sort at least some of the identifiedsubsets of items, in dependence on at least one of the one or morecalculated metrics, to produce a sorted subsets of items. Theprocessor(s) 112 can then use one or more feature selection process(es)to select for display, from the sorted subsets of items, information(e.g., item attributes) about two or more of the identified subsets ofthe items in the inventory, as well as where and/or how such informationis displayed, as will be described in additional detail below. Theprocessor(s) 112 can also use one or more feature selection process(es)to select which client attributes are displayed to an entity 132 that istasked with selecting items for a client, as well as where and/or howsuch client attributes are displayed, as will be described in additionaldetail below.

In accordance with an embodiment, the information about two or more ofthe identified subsets of the items in the inventory can be displayedvia a communication interface (e.g., a web browser) to an entity(represented by block 132 in FIG. 1) that is tasked with selecting itemsfrom the item inventory for one of the clients. This eliminates the needfor any of the entities (represented by blocks 132 in FIG. 1) to obtaincustom software to enable them to review recommended subsets of items(produced by the processor(s) 112) in order to perform their job ofselecting one or more items from the item inventory to be provided(e.g., mailed, shipped, sent, transmitted or otherwise provided) to aclient. In other words, each entity 132 can utilize a communicationsinterface (e.g., a web browser) and a communication network (e.g., theInternet, a LAN, a WAN, a wireless network, an intranet, a privatenetwork, a public network, a switched network, combinations of these,and/or the like) to view subsets of items that are recommended for aparticular client. In an alternative embodiment, the entities(represented by blocks 132 in FIG. 1) may use customized software toenable them to review recommended subsets of items (produced by theprocessor(s) 112) in order to perform their job of selecting one or moreitems from the item inventory to be provided (e.g., mailed, shipped,sent, transmitted or otherwise provided) to a client.

More specifically, in accordance with certain embodiments, theprocessor(s) 112 can execute the executable code for at least one of thefeature selection process(es) stored in the database 122-4 to therebyselect features for presentation to one of the entities 132 that istasked with selecting items from the item inventory for the client 142.The feature selection process(es), which can also be referred to asattribute selection process(es), may specify which client attributes fora client 142 are displayed on a screen to an entity 132 that is taskedwith selecting items for the client 142, as well as where and/or how theclient attributes for the client 142 are displayed on the screen. Thefeature selection process(es) may also specify which item attributes(for items identified for the client 142 by two or more recommendationprocesses) are displayed on a screen to the entity 132 that is taskedwith selecting items for the client 142, as well as where and/or how theitem attributes are displayed on the screen. With respect to “where”client or item attributes may be displayed on a screen, this can refer,e.g., to locations at which attributes are displayed, such as in the topleft corner, the top right corner, the bottom left corner, the bottomright corner, the top center, the bottom center, along a left side,along a right side, in a specific column, in a specific row, etc. Withrespect to “how” client or item attributes may be displayed, this canrefer, e.g., to the size of images that are displayed, the color and/orfont size of text that is displayed, etc.

FIG. 2 will now be used to illustrate an example of the information thatcan be displayed to one of the entities 132 tasked with selecting itemsfrom the item inventory for one of the clients 142. For example, theinformation shown in FIG. 2 can be presented via a web page that isaccessible via a web browser, but is not limited thereto. The specificinformation displayed in FIG. 2 and/or the location of the informationdisplayed in FIG. 2 can be determined using at least one featureselection process, the executable code for which has been described asbeing stored in the database 122-4, but can be stored in anotherdatabase, or more generally, in any data store.

Referring to FIG. 2, certain client attributes (which can also bereferred to as client features) for a specific client are shown in theleft most column 212, including: name, age, city and state of residence,bra size, top size, dress size and pant size. Other client attributes(which can also be referred to as client features) included in the leftmost column 212 are information specifying to what extent the clientlikes specific styles, information specifying how much the client iswilling to spend on various types of items of apparel (which informationcan be referred to as spendiness), and historical information specifyingto what extent the client has a preference for specific designers orbrands. Another client attribute that may be displayed, but is not shownin FIG. 2, is a picture of the client. To the right of the left mostcolumn 212 are a number of rows 222-1, 222-2, 222-3, 222-4, 222-5 and222-6, each of which can be referenced to as a row 222, and whichcollectively can be referenced as rows 222. Each of the rows 222includes information about a different subset of items identified, bythe processor(s) 112, using a different one of the recommendationprocesses included in the database 122-4. Such displayed informationabout items (which can also be referred to as item attributes or itemfeatures) can be textual and/or pictorial information. For example, inFIG. 2, each of the rows 222 includes thumbnails of six items in anidentified subset of items. In an embodiment, additional pictures and/orother details about an item can be accessed by clicking on or otherwiseselecting the thumbnail corresponding to the item. Each of the rows 222is also shown as including the name of the theme of the recommendationprocess that was used to select the subset of items included in the row.For example, the theme associated with the row 222-1 is “great for TX”,the theme associated with the row 222-2 is “great for tall”, the themeassociated with the row 222-3 is “great for busty”, the theme associatedwith the row 222-4 is “great for glam” (where “glam” is short forglamourous), the theme associated with the row 222-5 indicates thatclients that bought a particular brand (loved by this particular client)also bought . . . , and the theme associated with the row 222-6 is“tried & true”. Also shown in FIG. 2 are buttons 232 that enable anentity to filter the displayed items, e.g., such that if the “tops”button was selected, only tops would be displayed or highlighted in therows 222.

In accordance with an embodiment, the processor(s) 112 can use a rowordering type or aspect of a feature selection process to determine theorder of the rows 222 shown in FIG. 2. Such a process can use metrics,examples of which were mentioned above, to order the rows 222. A furthertype or aspect of a feature selection process, such as a relevancyprocess, can be used to specify the order of the items in each row,e.g., so a most relevant item (of an identified subset of items) for aparticular client is shown at the left most position in the row, and theleast relevant item is shown at the right most position in the row.Other variations are also possible, and within the scope of theembodiments described herein.

In FIG. 2, items identified using the same recommendation process aredisplayed within a same row, i.e., each row corresponds results producedusing a same one of the recommendation processes. Alternatively, itemsidentified using the same recommendation process can be displayed withina same column, i.e., each column can correspond to one of therecommendation processes. More generally, inventory determined to beappropriate for a client can be divided into facets, with each facetrepresenting a subset of items that is matched to zero, one or morespecific client attributes.

Referring back to FIG. 1, information about which subsets of items andwhich attributes of the items are actually displayed to the entities 132tasked with selecting items for clients 142 can be provided to thefeedback interface 152, which information can be referred to as coveragefeedback. Additional types of feedback that can be collected andutilized include where individual items were presented within a row orcolumn, which rows or columns did the entity (tasked with selectingitems for a client) select from, which rows or columns were ignored.Such feedback information can be used, e.g., to calibrate thequalification, selection and/or sales rate for each subset of items,and/or the items therein. Additionally, information about which itemswere selected by the entities 132 to provide to clients 142 can also beprovided to the feedback interface 152, which information can bereferred to as selection feedback. More specifically, selection feedbackcan include indications of specific ones of the items that are selectedfrom the item inventory for a client 142 by the one of the entities 132that is tasked with selecting items from the item inventory for theclient 142. In accordance with specific embodiments, the coveragefeedback and/or the selection feedback can be used to modify one or morefeature selection process(es) to improve the performance of the overallrecommendation system 102.

After items are provided (e.g., mailed, shipped, sent, transmitted orotherwise provided) to clients 142, each client determines what itemsthey will keep or otherwise accept and which items they will return orotherwise reject (i.e., not accept). In certain embodiments, a clientmay need to pay for (i.e., purchase) the items that they accept, butneed not pay for the items that they do not accept. Where items areshipped to clients, the clients keep the items that they accept and payfor, and the clients return the items they do not accept and do not payfor. Information about which items particular clients accept and whichitems particular clients reject (i.e., do not accept) is also shown asbeing provided to the feedback interface, which information can bereferred to as client feedback. More generally, client feedback data caninclude information obtained explicitly or implicitly from a clientabout one or more of the items provided to a client 142 after theitem(s) were selected from the item inventory for the client 142 by theone of the entities 132 that is tasked with selecting items from theitem inventory for the client 142. The client feedback can be entered bya client and/or can be entered by some other entity, e.g., an entitythat is responsible with tracking returned or otherwise unaccepteditems. Other variations are also possible. For example, a client mayprovide explicit subjective feedback that specifies to what extent theywere satisfied and/or not satisfied with particular items provided tothem. Additionally, or alternatively, in certain embodiments clients canbe asked to respond to survey questions, with answers to such questionbeing another example of explicit subjective client feedback. Moregenerally, a system (e.g., 102) can prompt a client (e.g., 142) toprovide explicit subjective feedback for various different attributesassociated with items selected for and provided to the client. Othertypes of client feedback can be considered implicit, in that it is basedon one or more actions made by a client, e.g., whether or not a clientaccepted or returned (and thus rejected) an item. Client feedback may beboth explicit and implicit, meaning both types of client feedback may beobtained. In accordance with certain embodiments, client feedback issaved as client history data in the client database 122-1.

The feedback interface 152 can provide the feedback it receives, orinformation indicate thereof, to the processor(s) 112. More generally,the processor(s) 112 have access to feedback received by the feedbackinterface 152 either directly from the feedback interface 152 or byaccessing the data store(s) where the feedback has been saved. Inaccordance with certain embodiments, the processor(s) 112 obtain clientfeedback information (e.g., from the feedback interface 152) and employsadaptive machine learning to update or otherwise modify (in dependenceon the client feedback information) the client information and/or theitem information stored in the databases 122-2 and/or 122-3. Forexample, a client, via a user interface, may have initially indicatedthat they like apparel having a glamorous style, and dislike apparelhaving a casual style. However, despite the client's own opinion oftheir style preferences, the client may consistently purchase orotherwise accept apparel sent to them having a casual style, andconsistently return or otherwise reject apparel sent to them having aclassic style. In such a case, the client may have unintentionallymischaracterized their actual style preferences. Through such clientfeedback received by the feedback interface 152, the processor(s) 112may modify information stored within the client database 122-2 thatspecifies the style preferences of a client. For another example, if anitem indicated as being “good for tall” women is repeatedly not acceptedby tall women, but is repeated accepted by average height women, thenthe processor(s) 112 may modify one or more attributes of that item sothat the item is no longer associated with the attribute “good fortall”, but rather, has the attribute “good for average height”. Theclient information that is updated or otherwise modified based on clientfeedback can be client history information, which may be indicativewhich items were purchased or otherwise accepted by the client, whichitems were returned or otherwise not accepted by the client, how much aclient has actually spent on specific types of items, and/or the like.These are just some examples of the types of client history data thatmay be updated base on client feedback, which examples are not intendedto be all encompassing.

As mentioned above, the processor(s) 112 can calculate one or moremetrics that quantify one or more aspects of a subset of items.Exemplary metrics that quantify aspects of a subset of items include,but are not limit to, a metric indicative of probability of successfully(for making a sale), a metric indicative of profit, a metric indicativeof similarity to previous items accepted by a client, a metricindicative of dissimilarity to previous items accepted by a clientand/or the like. Such metrics can be determined, using metric processes,in dependence on client information stored in the client database 122-1and/or item information stored in the inventory database 122-2. Inaccordance with an embodiment, the processor(s) 112 can modify, independence on the client feedback, how one or more such metrics arecalculated.

Attributes specified based on human judgment, by a client and/orexpert-human, can be fallible, in contrast to empirical data. Inaccordance with specific embodiments, attributes specified by a humancan be adjusted as empirical data is collected using the feedbackinterface 152. For example, a client may specify that they are “very”preppy, yet their purchase behavior may suggest that they are notpreppy, but rather are very classic. The client attribute information,entered or otherwise specified by the client, can represent a priorvalue indicative of a client's preference. Stored information specifyingthe client's preferences can be modified over time as more and moreempirical data is collected. For example, initially client specifiedattributes can be weighted very highly, and thereafter, as moreempirical data is collected the client specified attributes can beweighted less and less, and the empirical data can be weighted more andmore. Similar adjustments can be made to item attributes initiallyspecified by human experts.

In accordance with certain embodiments, the processor(s) 112 modify oneor more feature selection process(es) that is/are stored in the database122-4, based on the updated stored client history data, in order toimprove performance of the overall system 102. As noted above, thefeature selection process(es) is/are used to select features (alsoreferred to as attributes) that are presented to the entities 132 thatare tasked with selecting items from the item inventory for the clients142. This may include not only which attributes of recommended items areselected for presentation to an entity 132 tasked with selecting itemsfor a client 142, but also “where” item attributes are displayed (e.g.,such as in the top left corner, the top right corner, the bottom leftcorner, the bottom right corner, the top center, the bottom center,along a left side, along a right side, etc.) and “how” item attributesare displayed (e.g., such as the size of images that are displayed, thecolor and/or font size of text that is displayed, etc.). Similarly, thismay include not only which attributes of the client 142 are selected forpresentation to the entity 132 tasked with selecting items for a client142, but also “where” client attributes are displayed (e.g., such as inthe top left corner, the top right corner, the bottom left corner, thebottom right corner, the top center, the bottom center, along a leftside, along a right side, etc.) and “how” item attributes are displayed(e.g., such as the size of images that are displayed, the color and/orfont size of text that is displayed, etc.). More generally, inaccordance with certain embodiments, one or more feature selectionprocess(es) can be modified based on client history data, clientfeedback, selection feedback and/or coverage feedback, in order toimprove performance of the system 102. System performance can be ameasure of sales, client satisfaction, client retention, profitability,or combinations thereof, but is not limited thereto.

The high level flow diagram of FIG. 3 will now be used to summarizemethods according to various embodiments of the present technology. Suchmethods provide a mechanism to improve the performance of a system thatutilizes a combine machine-learning processes with expert-human judgmentto select items from an item inventor for clients. Referring to FIG. 3,step 302 involves storing client attribute data and client history datafor each of a plurality of clients. Such client attribute data andclient history data can be stored, for example, in the client database122-1 discussed above with reference to FIG. 1. Step 304 involvesstoring item attribute data for each of a plurality of items included inan item inventory. Such item attribute data can be stored, for example,in the inventory database 122-2 discussed above with reference toFIG. 1. Step 306 involves storing entity attribute data for each of aplurality of entities that may be tasked with selecting items from theitem inventory for one the plurality of clients. Such entity attributedata can be stored, for example, in the entity database 122-3 discussedabove with reference to FIG. 1. Step 308 involves storing executablecode for each of a plurality of different recommendation processes thatare each executable to identify, for any particular one of the clients,a subset of the items in the item inventory that is recommended for theparticular one of the clients. Step 310 involves storing executable codefor one or more feature selection process(es) that is/are executable toselect which of the item attributes and which of the client attributesare to be presented to one of the entities the may be tasked withselecting items for one of the clients from the item inventory. Theexecutable code stored at steps 308 and 310 may be stored, for example,in the recommendation and feature selection process database 122-4discussed above with reference to FIG. 1. Additional details of the dataand executable code stored at steps 302, 304, 306, 308 and 310 weredescribed above with reference to FIG. 1, and thus, need not berepeated. The data and executable code stored at steps 302, 304, 306,308 and 310 can alternatively be stored in more or less databases thandescribed (and shown in FIG. 1) and/or in other types of data stores.

Still referring to FIG. 3, steps 312 and 314, which can be performed byone or more processors (e.g., 112 in FIG. 1), are performed for eachclient of two or more of the clients. Step 312 involves executing theexecutable code for two or more of the plurality of recommendationprocesses to thereby identify a plurality of different subsets of theitems in the item inventory that are recommended for the client. Step314 involves executing the executable code for at least one of the oneor more feature selection process(es) to thereby select features forpresentation (e.g., display), to one of the entities that is tasked withselecting items from the item inventory for the client. In accordancewith specific embodiments, the selected features include informationabout two or more item attributes (associated with items includes in twoor more of the plurality of different subsets of the items in theinventory that are identified as a result of the executable code for twoor more of the plurality of recommendation processes being executed),and information about two or more of the client attributes (associatedwith the client). Details of various types of recommendation processesand feature selection process(es) were described above with reference toFIGS. 1 and 2, and thus, need not be repeated.

Still referring to FIG. 3, steps 316 and 318, which can be performed bya feedback interface (e.g., 152 in FIG. 1), are performed for eachclient of the two or more of the clients. Step 316 involves receivingselection feedback data about one or more of the items that are selectedfrom the item inventory for the client by the one of the entities thatis tasked with selecting items from the item inventory for the client.Step 318 involves receiving client feedback data about the one or moreof the items selected from the item inventory for the client by the oneof the entities that is tasked with selecting items from the iteminventory for the client. In other words, at step 316 information aboutwhat items were selected by an entity 132 for a client 142 can beobtained, and at step 318, information about whether the client 142accepted or rejection the items selected and provided to them can beobtained. Further types of information can be obtained at steps 316 and318. Details of various types of feedback that can be received, e.g., bythe feedback interface 152 in FIG. 1, were described above withreference to FIGS. 1 and 2, and thus, need not be repeated. In betweensteps 316 and 318, items selected for a client 142 by an entity 132 areprovided to the client 142, at which point the client decides whichitem(s), if any, they accept, and which item(s), if any, they do notaccept.

Still referring to FIG. 3, step 320 involves updating the stored clienthistory data based the client feedback data. Step 322 involves modifyingat least one of the one or more feature selection process(es), thatis/are used to select features for presentation, based on the updatedstored client history data. In accordance with certain embodiments, step322 includes modifying which attributes of recommended items areselected for presentation to an entity 132 tasked with selecting itemsfor a client 142. Additionally, or alternatively, step 322 can includemodifying “where” item attributes are displayed (e.g., such as in thetop left corner, the top right corner, the bottom left corner, thebottom right corner, the top center, the bottom center, along a leftside, along a right side, etc.). Additionally, or alternatively, step322 can include modifying “how” item attributes are displayed (e.g.,such as the size of images that are displayed, the color and/or fontsize of text that is displayed, etc.).

In accordance with certain embodiments, step 322 includes modifyingwhich, where and/or how attributes of a client (which can also bereferred to as client attributes) are presented (e.g., displayed) to anentity 132 tasked with selecting items for a client 142. Exemplaryclient attributes that may be presented to an entity 132 that is taskedwith selecting items for a client 142, include: date of birth and/orage, height, weight, waist size, chest size, bra size, inseam length,torso length, shirt size, dress size, pant size, residential address,whether or not the client is a mother, occupation, and/or the like, apicture of the client, the client's preferred style (e.g., classic,preppy, casual, bohemian, etc.), preferred fit (e.g., loose, fitted,tight, etc.), the client's preferred color scheme (e.g., bright,subdued), the client's preferred patterns (e.g., solids, stripes,floral), the client's preferred and/or disliked colors, the client'sbody type (e.g., straight, pear, spoon, hourglass, top hourglass,inverted triangle, oval, diamond, etc.), how long the particular clienthas been a member of the recommendation service provided by the system102, how often items have been shipped to the client, which items werepurchased or otherwise accepted by the client, which items were returnedor otherwise not accepted by the client, how much a client has actuallyspent on specific types of items, and/or the like.

Certain embodiments of the present technology can be used to determinewhich client attributes, when presented to an entity 132 tasked withselecting items for a client 142, improve system performance or diminishsystem performance. For example, such embodiments can be used todetermine whether displaying a picture of the client 142, to the entity132 tasked with selecting items for the client 142, results in theentity 132 selecting items that are more to the client's liking, or lessto the client's liking. For another example, such embodiments can beused to determine whether displaying the age of the client 142, to theentity 132 tasked with selecting items for the client 142, results inthe entity 132 selecting items that are more to the client's liking, orless to the client's liking. Determinations of whether an item is moreor less to a client's liking can be based in client feedback, including,but not limited to, whether or not the client accepted or rejected theitems, and/or answers to questions presented to the client about itemsprovided to the client. More generally, such embodiments can be used todetermine which client attributes when presented, improve systemperformance, and which client attributes when presented, diminish systemperformance. Such embodiments can additionally, or alternatively, beused to determine whether certain modifications to where and/or howclient attributes are presented, improve system performance or diminishsystem performance.

In accordance with certain embodiments, step 322 includes modifyingwhich, where and/or how item attributes are presented to an entity 132tasked with selecting items for a client 142. Exemplary item attributesthat may be selected for presentation to an entity 132 tasked withselecting items for a client 142, include: item type (e.g., pant, skirt,dress, etc.), item designer and/or brand, item size, item color, itemSKU number, item pattern (e.g., solid, checked, pinstripe, etc.), itemmaterial (e.g., cotton, wool, synthetic, etc.), good for tall women,good for short women, good for busty women, good for women in the Stateof Texas, good for preppy women, good for women in their 20s, good forwomen in their 50s, good for women with a long torso, casual, dressy,good for tropical vacations, good for spring, good for winter, fits trueto size, runs small, runs large, good for women with a specific bodytype (e.g., straight, pear, spoon, hourglass, top hourglass, invertedtriangle, oval, diamond, etc.), item sale price, item profit margin (ifpurchased by a client), how many units of an item are presently in theitem inventory, how often an item has been accepted (e.g., purchased) byclients, how often an item once provided to clients has been accepted(e.g., purchased) by clients, and/or the like.

Certain embodiments of the present technology can be used to determinewhich item attributes, when presented to an entity 132 tasked withselecting items for a client 142, improve system performance or diminishsystem performance. For example, such embodiments can be used todetermine whether displaying an item designer and/or brand, to theentity 132 tasked with selecting items for the client 142, results inthe entity 132 selecting items that are more to the client's liking, orless to the client's liking. For another example, such embodiments canbe used to determine whether displaying that an item is dressy orcasual, to the entity 132 tasked with selecting items for the client142, results in the entity 132 selecting items that are more to theclient's liking, or less to the client's liking. As noted above,determinations of whether an item is more or less to a client's likingcan be based in client feedback, including, but not limited to, whetheror not the client accepted or rejected the items, and/or answers toquestions presented to the client about items provided to the client.More generally, such embodiments can be used to determine which itemattributes when presented, improve system performance, and which itemattributes when presented, diminish system performance. Such embodimentscan additionally, or alternatively, be used to determine whether certainmodifications to where and/or how item attributes are presented, improvesystem performance or diminish system performance.

Details of how the feature selection process(es), stored in the database122-4, may be modified in accordance with certain embodiments, will nowbe described with reference to the flow diagram of FIG. 4. Referring toFIG. 4, step 402 involves systematically changing which features (e.g.,client attributes and/or item attributes) are selected by the featuresselection process(es) for presentation at different instances thatfeatures are selected for presentation. Step 404 involves identifyingwhich of the features when presented result in the items being selectedby the entities for the clients receiving positive client feedback, andwhich of the features when presented result in the items being selectedby the entities for the clients receiving negative client feedback. Step406 involves modifying at least one of the one or more feature selectionprocess(es) so that one or more of the features that are identified asresulting in positive client feedback are thereafter selected forpresentation or are more prominently presented, and one or more of thefeatures that are identified as resulting in negative client feedbackare thereafter not selected for presentation or are less prominentlypresented. If a client accepts an item, that can be interpreted aspositive client feedback, and if a client does not accept an item, thatcan be interpreted as negative client feedback. Client feedback may alsobe obtained based on a client's responses to questions presented to theclient. Other variations are also possible, and within the scope of theembodiments described herein.

The steps described with reference to FIG. 4 can be performed over timeas various entities 132 select items that are actually provided toclients 142. Alternatively, or additionally, the steps described withreference to FIG. 4 can be performed as part of simulations that usepreviously obtained client history data to simulate whether certainmodifications to feature selection process(es) will improve systemperformance or diminish system performance. For example, wheresufficient client history data is stored for a number of clients 142,simulations can be used to test whether showing pictures of the clients142, to entities 132 that may be tasked with selecting items for theclients 142 (during the simulations), help or hinder the entities 132 inselecting items that the clients 142 will like and accept. For anotherexample, where sufficient client history data is stored for a number ofclients 142, simulations can be used to test whether displaying theclients' age, to entities 132 that may be tasked with selecting itemsfor the clients 142 (during the simulations), help or hinder theentities 132 in selecting items that the clients 142 will like andaccept. For still another example, where sufficient client history datais stored for a number of clients 142, simulations can be used to testvarious different locations where certain client data may be displayedto entities 132, to see which location results in the entities 132making the best selections for the clients 142. The simulations can alsobe used to test out different locations (where) and/or ways (how) thatitem attributes and/or client attributes may be displayed to entitiesthat may be tasked with selecting items for the clients. Referring backto FIG. 1, such simulations can be performed, e.g., using a traininginterface 162. The training interface 162 can be used to train thefeature selection process(es), as well as to train the actual entities132 that may be tasked with selecting items from an item inventory forclients 142. The training interface 162 can be accesses using a webbrowser, or can be implemented using custom software. Entities 132 thatuse the training interface 162 may be presented with information similarto what is shown in FIG. 2. The types of client attribute data and/oritem attribute data that are displayed, as well as where and/or howdifferent client attribute data and/or item attribute data is displayed,can be systematically changed to determine what improves, and preferablyoptimizes, over system performance.

The embodiments of the present technology described above with referenceto FIGS. 1-4 can be used to intelligently, and in some instancessystematically, analyze big data related to clients and item inventory,and select which subsets of data should be presented to provide for goodoverall system performance. Such embodiments utilize coverage feedback,selection feedback and/or client feedback to improve the overall systemperformance over time. According, such embodiments of the presenttechnology address and at least partially overcome big data problemrelated to the visualization of big data that includes client data anditem inventory data.

Referring back to FIG. 1, in accordance with certain embodiments, theprocessor(s) 112 can access client history data to produce trainingtasks for the entities 132 that may be tasked with selecting items forthe clients 142 from the item inventory, and the training tasks can bepresented to at least some of the entities 132 for the purpose oftraining the entities 132 and/or training one or more feature selectionprocess(es). During such training, which can also be referred to assimulations, training feedback can be provided to the entities 132 totell them whether the items they selected would, or would not have been,accepted by certain clients, and in certain instances, why. For example,the training feedback can provide indications, based on the clienthistory data, regarding whether particular ones of the clients 142, forwhich the entities 132 selected items from the item inventory inresponse to the training tasks, had provided positive or negativefeedback regarding the items. Such embodiments can be used to train newentities 132, train poorly performing entities 132, or provide trainingto entities 132 that may in the future be matched with different typesof clients 142 than they had previously worked with. In accordance withcertain embodiments, the entity attribute data stored in the database122-3 can include training history data indicative of types of trainingprovided to specific entities 132 and/or the efficacy of the trainingprovided to specific entities 132. Such training history information canbe used, e.g., to assist in assigning tasks to the entities 132, whereinsuch tasks relate to selecting items from the item inventory for one ofthe clients. Such assignments can be performed, e.g., using a taskassignment process. More specifically, executable code for a taskassignment process can be stored in a data store, e.g., in the database122-4, and the processor(s) 112 can executed the code for the taskassignment process in order to assign tasks to entities 132.

Certain embodiments of the present technology, described herein, wereconceived from the need to measure, train, and understand the humancontribution to a recommendation system. While the development,training, testing, and optimization of traditional recommendationsystems may be a well-established process, best methods for workinghumans into that loop have not yet been specified. That said, humans canimprove recommendation systems by dealing with the types of data thatcomputers currently struggle with (e.g., images, text or otherunstructured data). Certain embodiments of the present technologydescribed herein provide a platform to study the human contribution tothe broader recommendation system, and then optimize their performanceto create a better overall recommendation system.

Traditionally, recommendation systems were developed by giving analgorithm a subset of training data, allowing it to make predictionsafter being trained, checking the outcomes of those recommendations,then feeding those outcomes back into the system to allow it to learn.Where humans (i.e., the entities 132) are part of a recommendationsystem, they can be trained in a similar way. To the point, the entities132 may receive a long list of recommendations produced using computerimplemented recommendation processes, and they are shown a similarfeature set about a given client, and are then asked to curate a shortlist of recommendations that are ultimately provided (e.g., sent) to theclient. Embodiments of the present technology described herein providesfor an understanding as to how the entities 132 go about making thatshort list, and then alters their behavior when needed in order toimprove their performance, and therefore the performance of the overallrecommendation system 102.

As described in additional detail above, embodiments described hereincan be used to determine both what information the entities 132 use andhow they make decisions based on it. This is important for understandingthe current configuration of the human component to the recommendationsystem 102, including which features the humans are using in order tomake their decisions. It can be thought of analogously to featureselection in algorithm development. For an example, embodiments of thepresent technology can be used to examine how the entities 132 use a setof displayed ‘style scores’ to select items for clients 142. Morespecifically, such embodiments have been used to find that the entities132 performed best (i.e., were most accurate) when given two sets ofthese scores (one for a client and one for the item they wereconsidering for that client). It is now believed that the entities 132likely compare the two sets of values to create a proxy for “similarity”between client and item. It is important to note that this was not userinterface (UI) testing. Rather, the study showed that the entities 132consider the information in the scores to making fast, on-the-flycalculations between those scores which increased their accuracy.

Using embodiments of the present technology, after studying a givenbehavior and drawing conclusions about what causes “better” outcomes,these same behaviors can be encouraged, thereby optimizing skills of theentities 132. Thus, embodiments described herein can also be used fortraining and behavior shaping. For example, during simulations,described above, the information displayed to entities 132 canexplicitly mimic some portion of the live recommendation process.Performance feedback can be provided immediately to enforce positivebehaviors and discourage negative ones without the normal lag (e.g., twoweeks) inherent in providing items to clients and waiting for clientfeedback. For example, assume it has been previously learned thatunderstanding how items fit different body types is crucial for makinggood recommendations. Based on this, a series of questions can becreated showing clients 142 with varying body types and asked entities132 whether a given garment would “fit” the client 142 or not. By usinghistorical client data to create the questions, this allows the systemto provide the entities 132 being trained with the correct responsesafter they tried to answer each question, and also enables the system toprovide the entities 132 with concept relevant feedback given by thatclient 142 immediately after the entity 132 answers a question.

Embodiments of the present technology can also be used to monitor theperformance of the entities 132, and to create highly rigorousexperiments which control variables, and directly measure abilities,thereby allowing their skills and weaknesses to be understood, andreactions in a more direct sense.

Logging all behaviors of the entities 132 when they are actuallyperforming tasks could be costly with regards to data storage. Forexample, it might be possible to log all mouse movements, clicks, andscrolls for all task related events, but this could require massive datastores and heavy processing. Certain embodiments described herein allowfor the collecting every type of data that is possible, but only asrequired. Embodiments described herein can also allow for an interfacewith other research technologies, such as eye trackers, to measurepsychophysical variables that could be difficult to interface withaccurately in a normal environment. This data can then be used to createquestions/trials for better understanding current behaviors of theentities 132, and then for creating recommendation training programs.

As mentioned above, there are well known techniques to train, test andtune computer implemented recommendation processes that are known to beeffective. Embodiments of the present technology described hereinexpands applications of such techniques to human entities. For example,embodiments described herein can be used to highlight information thatis useful to the entities 132, and remove extraneous information thatmight overload or slow down the entities 132.

Embodiments of the present technology described herein can be used toprovide highly targeted and personalized training regimens to improvethe entities 132 long term contribution to the system 102, therebyallowing the system 102 to improve its performance over time.

In many of the examples described above, the items in the item inventorywere described as being women's apparel, and the systems and methodswere described as employing adaptive machine learning to recommendsubsets of women's apparel to an entity that selects one or more piecesof women's apparel to send or otherwise provide to a client. However,embodiments of the present technology are not limited to such types ofitems, and more generally, can be used for recommending almost any typeof item that would benefit from combining machine-learning processeswith expert-human judgment. For example, the items in the item inventorycan alternatively be men's apparel, food, wine, beer, artwork, music ormovies, just to name a few. Depending upon the type of items, certaintypes of items that are selected for a client can be provided to aclient by delivering, mailing or shipping the items to the client, whileother types of items can be delivered electronically over wired orwireless communication networks. Preferably, the entity that is taskedwith selecting items for a client, based on recommendations generatedusing one or more computers, has an acceptable level of expertise in thetype of items included in the item inventory. For example, where theitems are women's apparel, the entity that select which items to be sentto a client may have studied fashion, worked in fashion, and/or workedin clothing retail. For another example, where the items are wine, theentity that selects which items are to be sent to a client may be asommelier, a wine collector or have worked for a vineyard. These arejust a few examples that are not intended to be all encompassing.

The items in an item inventory need not be physical products. Forexample, the items in an item inventory can alternatively be insurancepolicies. In other words, embodiments of the present technology can beused to select insurance policies for clients. For another example, theitems in an item inventory can be travel packages, in which caseembodiments of the present technology can be used to select travelpackages for clients. It is also possible that the items in an iteminventory are applicants for a job, in which case, embodiments of thepresent technology can be used to help sift through resumes, or thelike, to select specific prospective hires for clients. For stillanother example, the items in an item inventory can be studentapplicants, in which case, embodiments of the present technology can beused to help sift through school applications to select which studentapplicants should be accepted to a school. These are just a fewexamples, which are not intended to be all encompassing. More generally,embodiments described herein are especially useful for providingrecommendations in domains where: (1) there are numerous choicesavailable to select from for a client; (2) the products or services (ormore generally, the items) need to be personalized for a client; (3)there exists both structured and unstructured data about the itemsand/or clients, which would benefit from a combination ofmachine-learning processes and expert-human judgment (e.g., so machinescan handle the structured data, and expert-humans can handle theunstructured data); and (4) a client is given an opportunity to tryselected items and provide feedback regarding what they like and don'tlike about the items they tried. As noted above, the types ofrecommendation processes and the themes associated therewith can varygreatly, depending upon the types of items in an item inventory and theclients for which items are being selected.

The subject matter described herein may be embodied in systems,apparatus, methods, and/or articles depending on the desiredconfiguration. In particular, various implementations of the subjectmatter described herein may be realized in computer software, firmwareor hardware and/or combinations thereof, as well as in digitalelectronic circuitry, integrated circuitry, and the like. These variousimplementations may include implementation in one or more computerprograms that are executable and/or interpretable on a programmablesystem including at least one programmable processor, which may bespecial or general purpose, coupled to receive data and instructionsfrom, and to transmit data and instructions to, a storage system, atleast one input device, and at least one output device.

These computer programs (also known as programs, software, softwareapplications, applications, components, or code) include machineinstructions for a programmable processor, and may be implemented in ahigh-level procedural and/or object-oriented programming language,and/or in assembly/machine language. As used herein, the term“machine-readable medium” refers to any computer program product,apparatus and/or device (e.g., magnetic discs, optical disks, memory,Programmable Logic Devices (PLDs), but not limited thereto) used toprovide machine instructions and/or data to a programmable processor,including a machine-readable medium that receives machine instructionsas a machine-readable signal. The term “machine-readable signal” refersto any signal used to provide machine instructions and/or data to aprogrammable processor.

To provide for interaction with a user, the subject matter describedherein may be implemented on a computer having a display device (e.g., acathode ray tube (CRT) or liquid crystal display (LCD) monitor, but notlimited thereto) for displaying information to the user and a keyboard,touch screen and/or a pointing device (e.g., a mouse, touchpad or atrackball, but not limited thereto) by which the user may provide inputto the computer. Other kinds of devices may be used to provide forinteraction with a user, administrator and/or manager as well; forexample, feedback provided to the user, administrator and/or manager maybe any form of sensory feedback (e.g., visual feedback, auditoryfeedback, or tactile feedback); and input from the user may be receivedin any form, including acoustic, speech, or tactile input.

The subject matter described herein may be implemented in a computingsystem that includes a back-end component (e.g., as a data server), orthat includes a middleware component (e.g., an application server), orthat includes a front-end component (e.g., a client computer having agraphical user interface (GUI) or a Web browser through which a user mayinteract with an implementation of the subject matter described herein),or any combination of such back-end, middleware, or front-endcomponents. The components of the system may be interconnected by anyform or medium of digital data communication (e.g., a communicationnetwork). Examples of communication networks include, but are notlimited to, a local area network (“LAN”), a wide area network (“WAN”),and the Internet.

The computing system may include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

The implementations set forth in the foregoing description do notrepresent all implementations consistent with the subject matterdescribed herein. Instead, they are merely some examples consistent withaspects related to the described subject matter. Wherever possible, thesame reference numbers will be used throughout the drawings to refer tothe same or like parts.

While various embodiments of the present invention have been describedabove, it should be understood that they have been presented by way ofexample, and not limitation. It will be apparent to persons skilled inthe relevant art that various changes in form and detail can be madetherein without departing from the spirit and scope of the invention.For example, although a few variations have been described in detailabove, other modifications or additions are possible. In particular,further features and/or variations may be provided in addition to thoseset forth herein. For example, the implementations described above maybe directed to various combinations and sub-combinations of thedisclosed features and/or combinations and sub-combinations of severalfurther features disclosed above. In addition, the logic flow depictedin the accompanying figures and/or described herein do not require theparticular order shown, or sequential order, to achieve desirableresults. Other embodiments may be within the scope of the followingclaims.

Embodiments of the present invention have been described above with theaid of functional building blocks illustrating the performance ofspecified functions and relationships thereof. The boundaries of thesefunctional building blocks have often been defined herein for theconvenience of the description. Alternate boundaries can be defined solong as the specified functions and relationships thereof areappropriately performed. Any such alternate boundaries are thus withinthe scope and spirit of the claimed invention. One skilled in the artwill recognize that these functional building blocks can be implementedby discrete components, application specific integrated circuits,processors executing appropriate software and the like or anycombination thereof.

The breadth and scope of the present invention should not be limited byany of the above-described exemplary embodiments, but should be definedonly in accordance with the following claims and their equivalents.

1. (canceled)
 2. A system, comprising: one or more data storesconfigured to store: client attribute data that includes a plurality ofclient attributes and client history data, for each of a plurality ofclients, wherein the plurality of client attributes include one or moreclient specified attributes; and item attribute data that includes aplurality of item attributes, for each of a plurality of items includedin an item inventory; one or more processors coupled to the one or moredata stores and configured to: select, using one or more featureselection processes, from the plurality of item attributes an itemattribute and from the plurality of client attributes a client specifiedattribute associated with a client, wherein the selected item attributeand the selected client specified attribute are selected forpresentation to an entity tasked with selecting one or more items forthe client; select a subset of a plurality of different recommendationprocesses to use for the client; identify a plurality of differentsubsets of the items in the item inventory that are recommended for theclient using the subset of the plurality of different recommendationprocesses; provide to the entity tasked with selecting one or more itemsfor the client the plurality of different subsets of items in the iteminventory that are recommended for the client; receive selectionfeedback from the entity tasked with selecting one or more items for theclient and a selection feedback from the client, wherein the clientselection feedback indicates one or more items selected by the clientamong the one or more items that were selected by the entity, whereinthe client selection feedback is used to update the client history data;and modify the one or more feature selection processes based on theclient selection feedback, wherein a manner in which the one or morefeature selection processes select item attributes and client attributesto select items for the client are modified based on the clientselection feedback including by modifying a corresponding weightassociated with the selected client specified attribute based on theclient selection feedback.
 3. The system of claim 2, wherein the one ormore processors are further configured to: systematically change whichfeatures are selected by a feature selection process for presentation atdifferent instances that features are selected for presentation;identify which of the features when presented result in the items beingselected by one or more entities for clients receiving positive clientfeedback, and which of the features when presented result in the itemsbeing selected by the one or more entities for the clients receivingnegative client feedback; and modify the feature selection process sothat one or more of the features that are identified as resulting inpositive client feedback are thereafter selected for presentation or aremore prominently presented, and one or more of the features that areidentified as resulting in negative client feedback are thereafter notselected for presentation or are less prominently presented.
 4. Thesystem of claim 2, wherein each of the different recommendationprocesses utilize the client attribute data, the client history dataand/or the item attribute data in a different manner than otherrecommendation processes of the plurality of different recommendationprocesses.
 5. The system of claim 2, wherein a modification of the oneor more feature selection processes, based on the updated client historydata, results in information about item attributes that are selected forpresentation being modified to thereby improve performance of thesystem.
 6. The system of claim 2, wherein a modification of the one ormore feature selection processes, based on the updated client historydata, results in information about client attributes that are selectedfor presentation being modified to thereby improve performance of thesystem.
 7. The system of claim 2, wherein the one or more processors arefurther configured to: access the client history data to producetraining tasks for one or more entities that are tasked with selectingitems for the clients from the item inventory; present at least some ofthe training tasks to at least some of the one or more entities that aretasked with selecting items for the clients from the item inventory; andprovide training feedback to the one or more entities that are presentedwith at least some of the training tasks.
 8. The system of claim 7,wherein the training feedback provides indications, which are based onthe client history data, regarding whether particular ones of theclients, for which the one or more entities selected items from the iteminventory in response to the training tasks, had provided positive ornegative feedback regarding the selected items.
 9. The system of claim7, wherein: the one or more data stores are further configured to store:entity attribute data, for each of the one or more entities that aretasked with selecting items from the item inventory for one or more ofthe clients, the entity attribute data for each entity of at least someof the one or more entities including training history data indicativeof at least one of types of training provided to the entity or efficacyof the training provided to the entity; and a task assignment processthat is executable by the one or more processors to assign at least sometasks to at least some of the entities based on the training data storedfor the one or more entities, each of the tasks relating to selectingitems from the item inventory for one of the clients.
 10. The system ofclaim 9, wherein the one or more processors function as a taskassignment server when executing the task assignment process.
 11. Thesystem of claim 2, wherein the one or more processors are configured toperform simulations based on the client history data and modify at leastone of which, where, or how client attributes or item attributes aredisplayed based on results of the simulations.
 12. A method, comprising:selecting, by one or more processors, using one or more featureselection processes, an item attribute from a plurality of itemattributes and a client specified attribute associated with a clientfrom a plurality of client attributes, wherein the selected itemattribute and the selected client specified attribute are selected forpresentation to an entity tasked with selecting one or more items forthe client, wherein the one or more processors are coupled to a one ormore data stores, wherein the one or more data stores are configured tostore: client attribute data that includes the plurality of clientattributes and client history data, for each of a plurality of clients;and item attribute data that includes a plurality of item attributes,for each of a plurality of items included in an item inventory, whereinthe client attribute data includes the selected client attribute and theitem attribute data includes the selected item attribute; selecting, bythe one or more processors, a subset of a plurality of differentrecommendation processes to use for the client; identifying a pluralityof different subsets of the items in the item inventory that arerecommended for the client using the subset of the plurality ofdifferent recommendation processes; providing to the entity tasked withselecting one or more items for the client the plurality of differentsubsets of items in the item inventory that are recommended for theclient; receiving selection feedback from the entity tasked withselecting one or more items for the client and a selection feedback fromthe client, wherein the client selection feedback indicates one or moreitems selected by the client among the one or more items that wereselected by the entity, wherein the client selection feedback is used toupdate the client history data; and modifying the one or more featureselection processes based on the client selection feedback, wherein amanner in which the one or more feature selection processes select itemattributes and client attributes to select items for the client aremodified based on the client selection feedback including by modifying acorresponding weight associated with the selected client specifiedattribute based on the client selection feedback.
 13. The method ofclaim 12, further comprising: systematically changing which features areselected by a feature selection process for presentation at differentinstances that features are selected for presentation; identifying whichof the features when presented result in the items being selected by oneor more entities for the clients receiving positive client feedback, andwhich of the features when presented result in the items being selectedby the entities for the clients receiving negative client feedback; andmodifying at least one of the one or more feature selection processes sothat one or more of the features that are identified as resulting inpositive client feedback are thereafter selected for presentation or aremore prominently presented, and one or more of the features that areidentified as resulting in negative client feedback are thereafter notselected for presentation or are less prominently presented.
 14. Themethod of claim 12, wherein each of the different recommendationprocesses utilize the client attribute data, the client history dataand/or the item attribute data in a different manner than otherrecommendation processes of the plurality of different recommendationprocesses.
 15. The method of claim 12, wherein modifying at least one ofthe one or more feature selection processes includes modifyinginformation about item attributes that are selected for presentation tothereby improve performance of the method based on updated clienthistory data.
 16. The method of claim 12, wherein modifying at least oneof the one or more feature selection processes includes modifyinginformation about client attributes that are selected for presentationto thereby improve performance of the method based on updated clienthistory data.
 17. The method of claim 12, further comprising: accessingthe client history data to produce training tasks for the one or moreentities that are tasked with selecting items for the clients from theitem inventory; presenting at least some of the training tasks to atleast some of the one or more entities that are tasked with selectingitems for the clients from the item inventory; and providing trainingfeedback to the one or more entities that are presented with at leastsome of the training tasks.
 18. The method of claim 17, wherein thetraining feedback provides indications, which are based on the clienthistory data, regarding whether particular ones of the clients, forwhich one or more the entities selected items from the item inventory inresponse to the training tasks, had provided positive or negativefeedback regarding the items.
 19. The method of claim 17, furthercomprising: storing entity attribute data, for each of the one or moreentities that may be tasked with selecting items from the item inventoryfor one or more of the clients, the entity attribute data for eachentity of at least some of the one or more entities including traininghistory data indicative of at least one of types of training provided tothe entity or efficacy of the training provided to the entity; andassigning at least some tasks to at least some of the one or moreentities based on the training data stored for the one or more entities,each of the tasks relating to selecting items from the item inventoryfor one of the clients.
 20. The method of claim 12, wherein the one ormore data stores are further configured to store a task assignmentprocess, wherein the one or more processors function as a taskassignment server when executing the task assignment process.
 21. Acomputer program product embodied in a non-transitory computer readablemedium and comprising computer instructions for: selecting, by one ormore processors, using one or more feature selection processes, an itemattribute from a plurality of item attributes and a client specifiedattribute associated with a client from a plurality of clientattributes, wherein the selected item attribute and the selected clientspecified attribute are selected for presentation to an entity taskedwith selecting one or more items for the client, wherein the one or moreprocessors are coupled to a one or more data stores, wherein the one ormore data stores are configured to store: client attribute data thatincludes the plurality of client attributes and client history data, foreach of a plurality of clients; and item attribute data that includes aplurality of item attributes, for each of a plurality of items includedin an item inventory, wherein the client attribute data includes theselected client attribute and the item attribute data includes theselected item attribute; selecting, by the one or more processors, asubset of a plurality of different recommendation processes to use forthe client; identifying a plurality of different subsets of the items inthe item inventory that are recommended for the client using the subsetof the plurality of different recommendation processes; providing to theentity tasked with selecting one or more items for the client theplurality of different subsets of items in the item inventory that arerecommended for the client; receiving selection feedback from the entitytasked with selecting one or more items for the client and a selectionfeedback from the client, wherein the client selection feedbackindicates one or more items selected by the client among the one or moreitems that were selected by the entity, wherein the client selectionfeedback is used to update the client history data; and modifying theone or more feature selection processes based on the client selectionfeedback, wherein a manner in which the one or more feature selectionprocesses select item attributes and client attributes to select itemsfor the client are modified based on the client selection feedbackincluding by modifying a corresponding weight associated with theselected client specified attribute based on the client selectionfeedback.